Skip to content

Conversation

evertlammerts
Copy link
Collaborator

This pulls in almost all bugfix work that was done since 1.4.0. It's way too big, but this is where we are. Hopefully things slow down on v.1.4-andium after this.

Tishj and others added 30 commits September 15, 2025 14:41
This has annoyed me for some time, needing to remember the filter
pushdown logic is defined in `arrow_array_stream.cpp` which is
completely illogical

Cleaned it up, added a Makefile for `make format-main`, and extended the
import cache to get rid of some raw `py::_module::import` calls
evertlammerts and others added 29 commits September 24, 2025 14:07
Fixes duckdb#74 

This PR:
- exports the `adbc_driver_duckdb` package so you can use it after `pip
install duckdb`
- doesn't just skip the tests if the package can't be found
- limits the exported symbols to the only two that are ever needed:
`duckdb_adbc_init` and `PyInit__duckdb`:

❯ nm -g -C -P
.venv/lib/python3.11/site-packages/_duckdb.cpython-311-darwin.so | ag -v
' U '
        _PyInit__duckdb T 41900 0
        _duckdb_adbc_init T 1054e6c 0
Fixes duckdb#57 

- Created and curated new stubs
- Added py.typed
- Added mypy config
- Copied the typing and functional modules into sqltypes and func and deprecated the originals, to prevent shadowing issues
See duckdb#98 

We have our own Polars IO plugin to create lazy polars dataframes. We
try to push as many predicates down into DuckDB as we can, for which we
try to map Polars expressions (including datatypes) to SQL expressions.
To do this we depend on Polar's `Expr.meta.serialize`.

None of this is very stable. Polars IO source plugins are marked
`@unstable` and `Expr.meta.serialize` says "Serialization is not stable
across Polars versions".

In this case the problems seems to come from Polars requiring an
explicit scale to be set for decimals ([this
pr](pola-rs/polars#24542)). The serialized
format seems to have changed into:
```json
{
  "expr": {
    "Literal": {
      "Scalar": {
        "Decimal": [
          1,
          38, // This now includes the scale
          0
        ]
      }
    }
  },
  "dtype": {
    "Literal": {
      "Decimal": [
        38, // this was already there
        0
      ]
    }
  },
  "options": "Strict"
}

```

Interestingly, even if we explicitly set precision to e.g. 20, the
relevant part of the serialized expression looks as follows:
```json
{
  "expr": {
    "Literal": {
      "Scalar": {
        "Decimal": [
          1,
          38, // Still 38?
          0
        ]
      }
    }
  },
  "dtype": {
    "Literal": {
      "Decimal": [
        20, // This is correct
        0
      ]
    }
  },
  "options": "Strict"
}
```

This PR allows for both a 2 and 3 item list for decimals.
@evertlammerts evertlammerts merged commit 8c6767a into duckdb:main Oct 7, 2025
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants